The news: In the summer of 2021, OpenAI quietly shuttered its mulrobotics team, announcing that progress was being stifled by a lack of data necessary to train robots in how to move and reason using artificial intelligence. Now three of OpenAI's early research scientists say the startup they spun off in 2017, called Covariant, has solved that problem. They've unveiled a system that combines the reasoning skills of large language models with the physical dexterity of an advanced robot. How it works: The new model, called RFM-1, was trained on years of data collected from Covariant's small fleet of item-picking robots, as well as words and videos from the internet. Users can prompt the model using five different types of input: text, images, video, robot instructions, and measurements.

The general problem of planning feasible trajectories for multimodal robots is still an open challenge. This paper presents a whole-body trajectory optimisation approach that addresses this challenge by combining methods and tools developed for aerial and legged robots. First, robot models that enable the presented whole-body trajectory optimisation framework are presented. The key model is the so-called robot centroidal momentum, the dynamics of which is directly related to the models of the robot actuation for aerial and terrestrial locomotion. Then, the paper presents how these models can be employed in an optimal control problem to generate either terrestrial or aerial locomotion trajectories with a unified approach. The optimisation problem considers robot kinematics, momentum, thrust forces and their bounds. The overall approach is validated using the multimodal robot iRonCub, a flying humanoid robot that expresses a degree of terrestrial and aerial locomotion. To solve the associated optimal trajectory generation problem, we employ ADAM, a custom-made open-source library that implements a collection of algorithms for calculating rigid-body dynamics using CasADi.